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Abstract 

This study examined the effects of question prompts, knowledge integration prompts, and 
problem solving prompts, embedded in a Web-based learning environment in scaffolding 
preservice teachers’ conceptual understanding and problem solving in an ill-structured 
domain. A mixed-method study was employed to investigate the outcomes of students’ 
conceptual knowledge and ill-structured problem solving. Tlje quantitative results indicated 
that students who received knowledge integration prompts had significantly higher scores 
in overall problem solving performance, blit the same was not true for prompts focused 
on conceptual knowledge. Further, the qiialitative findings revealed the positive effects of 
knowledge integration prompts in facilitating students to make intentional efforts to identify 
and explain major concepts and their relationships that are necessary for solving the ill- 
structured problem. This study has implications for designing curricida in ill-defined domains 
that seek to integrate and promote the application of educational principles to real-world 
problems. (Keywords: knowledge integration, ill-structtired problem solving, scaffolding.) 

INTRODUCTION 

An important characteristic of experts is their extensive, well-organized 
knowledge in their disciplines, which allows them to flexibly and efficiently 
retrieve relevant knowledge (Bransford, Brown, & Cocking, 2000). Experts also 
have cognitive frameworks that support problem analysis and guide retrieval 
and application of past experiences to solve new problems (Beck, McKeown, 

& Grommell, 1989; Ericsson & Staszewski, 1989; Glaser & Chi, 1988; 
Hasselbring, Coin, & Bransford, 1987; Simon, 1980). However, novices do 
not possess such integrative knowledge frameworks, which leads to a tendency 
toward oversimplifying the complexity of the new knowledge, generating vague 
relationships between prior and new knowledge, and processing information 
superficially and mindlessly (Feltovich, Spiro, Coulson, & Feltovich, 1996; 
Pressley et ah, 1992). Yet in previous research, novices’ deficiencies in problem 
solving have been attributed to limitations in both domain and metacognitive 
knowledge (Brown, 1987). 

Many researchers have emphasized that knowledge integration is an 
important process in students’ science learning because it engages students 
to monitor, actively reflect, evaluate, and modify their own knowledge (e.g., 
Davis & Linn, 2000; Linn & Hsi, 2000). Due to the lack of recent empirical 
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studies focused on ill-structured problem solving in social science domains, 
it is critical for the researchers to build a theoretical framework on the earlier 
literature from cognitive science perspectives. From the few investigations on 
the effectiveness of knowledge integration from previous research in supporting 
conceptual knowledge and ill-structured problem solving in the social sciences, 
we believed that scaffolding strategies targeted at knowledge integration and 
problem solving could support students’ cognitive and metacognitive skills. 

In this study, we specifically examined the effects of two types of question 
prompts, namely knowledge integration prompts and problem solving prompts, 
to scaffold students’ conceptual knowledge and problem solving processes in an 
ill-defined domain. The curriculum context used for this study was educational 
measurement because it represented authentic and real-world problems in the 
practice and research of education. A Web-based learning environment was 
created that provided the information students needed within the discipline as 
well as learning scenarios to engage them in authentic problem solving activities 
and guide their learning process through scaffolding (Hannafin, Land, & 

Oliver, 1999; Jonassen, 1999). 

THEORETICAL FRAMEWORK 
Knowledge Integration 

Learning is an active process by which students construct their own 
knowledge in light of their existing knowledge and through a process of 
generation, integration, and transformation of their experiential world (Gao, 
Baylor, & Shen, 2005). Unfortunately, students often hold multiple conflicting 
views before learning new information and create their repertoire of views 
without reflecting on their existing knowledge. Students’ existing knowledge 
serves as an interpretative framework for knowledge integration because new 
knowledge is filtered through existing knowledge. Consequently, students’ 
existing knowledge plays an important role in understanding new information. 
Understanding also goes hand-in-hand with the construction of an integrated 
conceptual framework (Hewitt, 2002). For this reason, promoting an integrated 
conceptual framework that furthers understanding should be explicitly taught. 

Thus, Linn and her colleagues (2000) sought to design an instructional aid 
that promoted a more integrated student understanding of complex science 
concepts and processes. The Web-based Inquiry Science Environment (WISE) 
is one of the curriculum projects they created to help students develop more 
cohesive, coherent, and thoughtful accounts of scientific phenomena (Linn, 
Clark, & Slotta, 2003). WISE was guided by an instructional framework 
called scaffolded knowledge integration framework (SKI), derived from 
substantive and extensive research such as that on cognitive apprenticeships 
(Collins, Brown, & Newman, 1989). Significantly, a scaffolded knowledge 
integration framework requires students to reflect on their deliberately 
developed repertoire of models for complex phenomena, and to work toward 
expanding, refining, reconciling, and linking these models (Bell, 2002; Linn, 
1995). In an intervention study, Davis and Linn (2000) found that students 
who were encouraged to monitor their learning progress and identify new 
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connections among ideas showed greater integrated understanding of the 
science phenomena compared to students who only devoted their attention to 
the inquiry process. Apparently, students who only focus on inquiry without 
a theoretical base are less likely to develop a robust conceptual understanding 
than students who compared ideas, distinguished cases, identified the links and 
connections among notions, sought evidence to resolve uncertainty, and sorted 
out valid relationships, thus improving their knowledge integration (Davis & 
Linn, 2000). 

Likewise, Scardamalia and Bereiter (2003) indicated that we need to prepare 
for student learning as knowledge building. In a computer-based learning 
environment called Computer-Supported Intentional Learning Environments 
(CSILE), students were encouraged to clarify problem statements, develop 
theories, state difficulties in understanding certain issues, and summarize what 
they have learned; thus, students actively build their own knowledge bases 
(Scardamalia & Bereiter, 1996). Associations developed exclusively within the 
context of new material were less effective for knowledge building than those 
developed between new material and prior understanding. They concluded that 
instruction should support students’ knowledge to be integrated, as opposed to 
disjointed and static. 

As indicated, the features of a scaffolded knowledge integration instructional 
framework recognize students’ weaknesses by critically examining their own 
thoughts and the evidence bearing on them in both their informal thinking 
about everyday topics and their thinking within formal academic studies 
(Linn, 2005). The framework has been examined and implemented in science 
inquiry learning contexts, in which students were constantly asked to reflect to 
help them monitor thoughts and construct a coherent and robust conceptual 
understanding (Linn & Hsi, 2000), which is also called metacognition. 

Ill- Structured Problem Solving 

In contrast to well-structured problems commonly encountered in 
educational settings, ill-structured problems are the kinds of problems that 
students face routinely in everyday life (Jonassen, 2002). Ill-structured problems 
have vague and less-defined goals and unstated constraint information (Voss, 
Wolfe, Lawrence, & Eagle, 1991). These problems have no right or wrong 
concepts, rules, and principles for arriving at the solution and possess multiple 
solutions or may not have any definite solution at all (Butler & Thomas, 

1999). As a result, experts and novices also approach these problems differently 
(Bransford, Brown, & Cocking, 2000). The components and processes in 
experts and novices’ ill-structured problem solving are discussed below. 

The primary predictor of successful problem solving is domain knowledge 
(Murphy & Alexander, 2002). How much solvers know about a domain is 
important to understanding the problem and generating solutions. However, 
ill-structured problems cannot be solved simply by finding the information 
and following a constrained set of rules. In fact, that domain knowledge must 
be well-integrated in order to support ill-structured problem solving. Such 
“integratedness” also is described as structural knowledge. Structural knowledge 
is the knowledge of integrating domain knowledge into useful procedural 
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knowledge for solving domain problems (Jonassen, Beissner, & Yacci, 1993). 

In a study conducted in the Soviet Union, Voss and his associates (1986) 
found that the domain knowledge possessed by novices seemed to consist of 
bits and pieces of information that were not integrated and that ultimately 
impaired their problem solving processes. A series of additional investigations 
on expert and novice problem solving also indicated that experts were better 
problem solvers because their representations acquire more integrated domain 
knowledge of the problem and that helps them to construct a meaningful 
internal representation that can be manipulated (Wineburg, 1998). In brief, 
domain knowledge is not enough to solve ill-structured problems; it must be 
integrated as structural knowledge to enable problem representation and solve 
ill-structured problems. 

In addition to representational and selection complexities, ill-structured 
problems have no clear solutions and demand that problem solvers consider 
alternative goals as well as handle competing goals. This requires problem 
solvers to control and monitor the selection and execution of a solution 
process. In other words, they have to use metacognitive skills such as self- 
awareness of cognitive knowledge and self-regulation of cognitive processes and 
strategies during problem solving (Brown, 1987). King (1991) examined the 
effectiveness of self-questioning as a metacognitive strategy on students’ reading 
comprehension. She found that self-questioning promotes internal dialogue 
for systematically analyzing problem information and regulating execution of 
cognitive strategies (King, 1991). Likewise, Delclos and Harrington (1991) 
found that students who monitored their own problem solving processes tended 
to use more metacognitive strategies for completing the task. In sum, successful 
problem solvers use self-questioning and monitoring to gain access to and guide 
execution of strategies and to regulate use of strategies and problem solving 
performance (Lin, 2001). Although well-structured and ill-structured problems 
share similarities in some respects, ill-structured problems often require solvers 
to go beyond what is represented in the problem statement and consider 
alternatives using more metacognitive skills. Ge and Land (2004) synthesized 
past studies and created a model for ill-structured problem solving, including 
four processes: (a) representing problem(s), (b) generating and selecting 
solutions, (c) making justifications, and (d) monitoring and evaluating goals 
and solutions. 

Question Prompts as Scaffolding Strategies 

Merely presenting information generally will not cause students to develop 
accurate and integrative knowledge that fosters the understanding of pragmatic 
principle(s) and dynamic stances towards new knowledge. Therefore, 
intentional instructional supports should be made to elicit such knowledge 
building processes (Scardamalia & Bereiter, 1994). Students need to be 
prompted to think about new material in such a way that they transform the 
material, thus constructing new knowledge. Question prompts as instructional 
supports have been found to effectively promote students’ knowledge 
integration. For instance, the studies of King and Rosenshine (1993), King 
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(1994), and Davis and Linn (2000) used prompts to engage students in 
knowledge integration that furthered their knowledge acquisition. In a similar 
manner, merely asking students to solve problems without providing necessary 
instructional supports appears to decrease the possibility of accomplishing the 
ill-structured problem solving process. Ge and Land (2003) investigated the use 
of question prompts in facilitating students’ ill-structured problem solving and 
found that students who were prompted by questions made increased deliberate 
efforts to identify and seek relevant information in the problem. 

Different question prompts may serve different needs and purposes for 
students. Recently, researchers have started to investigate in depth different 
types of questions that promote students’ cognition and metacognition. For 
example, Davis (2003) examined science students’ reflection productivity 
through what she labeled as “generic” and “directed” prompts. “Generic” 
prompts entailed having students to simply stop and think, whereas directed 
prompts were more elaborate, providing students with hints or directions for 
reflective thinking. She found that the “generic” prompts promoted more 
productive reflective thinking than did the “directed” prompts. Therefore, 
prompt types make a difference. While researchers in science contexts have 
begun to pay attention to different types of question prompts, it also is 
important to focus attention on the research in social science contexts. 

Despite the justification for the use of question prompts to facilitate 
knowledge integration in science learning, the relationship between different 
types of questioning strategies and social science has not been sufficiently 
studied. Davis and Linn (2000) studied the effects of guided questions on 
metacognitive skills, knowledge integration, and problem solving. However, 
in Davis and Linn’s (2000) study, the problems were situated in science 
and the subjects were eighth graders. Ge and Land (2003) provided a list of 
questions that were mapped to ill-structured problem solving processes to guide 
undergraduate students in completing ill-structured problem solving tasks in 
information science and technology. Although Ge and Land (2003) found 
that question prompts had positive effect on students’ ill-structured problem 
solving, their study did not examine the students’ prior knowledge aside from 
self-reports on prior problem solving experience across different conditions. 

The students’ self-reports might be a questionable measurement of what 
students knew. Further, Ge, Chen, and Davis (2005) called for future research 
to compare the effects of different question prompts on students’ conceptual 
knowledge and ill-structured problem solving. 

PURPOSE OF THE STUDY 

The purpose of this study was to investigate the effects of (a) knowledge 
integration prompts, (b) problem solving prompts, and (c) both knowledge 
integration and problem solving prompts on scaffolding students’ conceptual 
knowledge and problem solving processes in an ill-structured task. The problem 
solving outcomes and processes investigated were (a) problem representation, (b) 
developing and evaluating solutions, and (c) monitoring and justifying a plan of 
action (Ge & Land, 2004). 
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The knowledge integration prompts in this study referred to a set of questions 
that prompted students to connect ideas, compare ideas, seek uncertainty 
evidence, transfer ideas, and summarize valid relationships. They also were 
intended to direct students’ attention to explain and understand how, why, and 
what. The problem solving prompts in this study were a set of questions with 
specific procedures for problem solving. 

The study examined the following questions: 

1. Does the use of knowledge integration prompts only, problem 
solving prompts only, or the combination of knowledge integration 
and problem solving prompts have an effect on students’ knowledge 
acquisition and problem solving outcomes in the domain of 
educational measurement in the Web-based learning environment? 

2. How does the use of different question prompts influence students’ 
conceptual knowledge and ill-structured problem solving? 

METHODOLOGY 

A mixed-method study was employed to investigate two research questions. 
The use of quantitative and qualitative methods helped the researcher to 
examine the results from different data sources and expand the scope and 
breadth of the study. The experimental study, designed to answer research 
question 1, was conducted to measure students’ conceptual knowledge and 
problem solving outcomes in an ill-structured task. The qualitative study 
consisting of analysis students’ ill-structured problem solving reports and 
follow-up interviews added depth and anecdotal evidence to findings from 
quantitative measures. 

Participants 

Participants in the experimental design were 5 1 undergraduate students (32 
female and 19 male) recruited from three class sections of an introductory 
course in the Department of Educational Psychology at a south central U.S. 
university. Of these, 1 5 also participated in the follow-up interview. Most 
of the students were sophomores and juniors. About 70% had taken at least 
one educational psychology class and 47% had not yet taken an educational 
measurement class. 

THE EXPERIMENTAL STUDY 
Design and Procedures 

Four versions of Web-based learning environments were created. A database 
was developed so that participants’ responses could be saved and retrieved. The 
5 1 participants were assigned to one of four Web-based learning environment 
study conditions: a) knowledge integration (KI) (n=13), b) problem solving 
(PR) (n=l4), c) the combination of knowledge prompts and problem solving 
prompts (KP) (n=13), and d) control condition (Control) (n=ll). Following 
the pretest, the participants were directed to read a set of instructional passages 
about educational measurement, particularly reliability and validity, and then 
solve an ill-structured problem in the discipline. The participants in the KI 
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and KP conditions were provided with and required to respond to a series 
of knowledge integration prompts after the presentation of the instructional 
passages. Their completed responses would be submitted, saved to the database, 
and displayed later. The participants in the PS and Control conditions simply 
read through the instructional passages. During the ill-structured problem 
solving, only the participants in the PS and KP conditions were provided 
with and required to respond to a series of problem solving prompts after the 
presentation of the case study. Again, their completed responses were submitted, 
saved to the database, and displayed on the next screen, where the participants 
could copy and paste their prior responses to be organized and edited into 
final solution reports. The participants in the KI and KP conditions could also 
retrieve their responses from the knowledge integration prompts. After solving 
the problem, the participants completed a posttest before exiting the study. 

Materials 

Prompt Types. Knowledge integration prompts were adapted from King 
(1994) and Tinn (1995) studies, which addressed critiquing, interpreting, and 
explaining key concepts. For example, “Explain why reliability and validity are 
important,” and “Go beyond what was covered in the instructional passages, 
summarize the purpose and the meaning of reliability and validity.” Problem 
solving prompts were generated to be parallel with the ill-structured problem 
solving processes proposed by Ge and Land (2004), for example, “What facts 
from this case suggest a problem?”, “Why is it occurring?”, and “What specific 
strategies do you want to suggest to the teacher to help him solve the problems 
that you have identified?” 

Pre/Posttests. A 19-item test about reliability and validity in educational 
measurement was developed to assess students’ conceptual knowledge of the 
instructional passages provided. The test was evaluated by the instructors of 
measurement and evaluation courses to validate that the items were adequately 
and appropriately represented. The test included three major items: terminology 
(seven items), comprehension (seven items), and application (five items) tests. 
The test had a split-half internal consistency .65 reliability coefficient. 

Ill- Structured Problem. The problem was generated from a real-world 
educational measurement problem with embedded reliability and validity 
principles. A rubric was developed to assess students’ problem solving outcomes 
on the ill-structured problem. Students’ problem solving outcomes were scored 
numerically based on their performances on (a) problem representation, (b) 
developing and justifying solution(s), and (c) monitoring and evaluating a plan 
of action. These coding schemes were chosen because they serve as indicators 
for the processes in solving ill-structured tasks (Ge & Land, 2004). Instructors 
from the educational psychology department and experts in ill-structured 
problem solving research also validated the rubric. Before grading, two raters 
reached a conceptual consensus on how to interpret the scoring rubrics through 
discussion and examples. Any discrepancies of assigned values were discussed 
among raters. Consequently, a high consensus was reached. A split-half internal 
consistency procedure was performed and found that the scoring rubric had .87 
reliability coefficient. 
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Data Analyses 

Quantitative data was analyzed first, followed by qualitative data. Multivariate 
analysis of variance (MANOVA) was conducted to examine the effects of 
different types of prompts on students’ conceptual knowledge and ill-structured 
problem solving outcomes. MANOVA was used for this study because it allows 
researchers to include multiple dependent variables and evaluate whether 
the means on a set of dependent variables vary across levels of factors (Green 
& Salkind, 2003). Wilks’s Lambda F( a = .03) was used in interpreting the 
multivariate test results. As shown by the results from the Levene’s Test, the 
assumption of equal variance was met at the .05 alpha level, and thus met the 
MANOVA testing assumption that the residual errors follow a multivariate 
normal distribution in the population. 

THE QUALITATIVE STUDY 
Procedures and Materials 

All 5 1 participants’ problem solving reports were retrieved from the database 
by the first author. Different from the rubric that was developed to score 
quantitatively, the researcher coded each report individually in how participants 
defined and identified the concepts in their problem solving reports. Following 
the completion of the experimental study, participants were selected based on 
their willingness to take part in audiotaped interviews. 

Data Analyses 

The analysis process involved reading and jotting marginal notes on the 
transcripts, identifying emerging patterns into categories, and drawing 
conclusions. For example, the first author read through each student’s problem 
solving report and highlighted the places where he or she identified key 
concepts and their relationships. Highlighted places were used to examine the 
participants’ conceptual knowledge in the context of question prompts or no 
prompts. The next procedure was to organize and display the emerging patterns 
so that comparison could be made across different conditions. The analyses 
of problem solving reports and follow-up interviews added reliability to the 
findings along with member checks for validity and the use of multiple coders 
for inter-rater reliability (Bogdan & Biklin, 1998). 

RESULTS 

Quantitative Outcomes 

To answer research question 1, “Does the use of knowledge integration 
prompts only, problem solving prompts only, or the combination of knowledge 
integration and problem solving prompts have an effect on students’ conceptual 
knowledge and problem solving outcomes in the domain of educational 
measurement in the Web-based learning environment?,” multivariate analysis 
of variance (MANOVA) was employed. The results of MANOVA on the effect 
of students’ conceptual knowledge did not reveal significant differences among 
conditions, Wilks’s Lambda (A) =.80, F (6, 92) = 1.80,/».05, r| 2 =.11, observed 
power =.65. Table 1 shows means and standard deviations for the conceptual 
knowledge posttests by groups. 
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Table 1: Means and Standard Deviations for the Conceptual Knowledge by 

Question Prompt Types 



Condition 

N 

Mean 

Std. Deviation 

Pretest 

KI 

13 

6.92 

2.18 


PS 

14 

6.79 

2.33 


KP 

13 

8.08 

1.89 


Control 

11 

7.46 

2.12 

Posttest 

Kl 

13 

11.08 

2.60 


PS 

14 

9.43 

1.95 


KP 

13 

10.69 

2.43 


Control 

11 

9.00 

2.45 


Note: The possible ranges of scores for pre/posttests are 0—19. 


Table 2: Means and Standard Deviations of Dependent Variables by 
Treatment Group 


Treatment Group 









Knowledg 

e 





Knowledge 
integration (Kl) 

Problem solving 

(PR) 

integration 
and problem 
solving (KP) 

Control 


DVs 

N 

M 

SD 

N 

M 

SD 

N 

M 

SD 

N 

M 

SD 

Problem 

representation 

13 

5.08 

1.94 

14 

5.64 

2.82 

13 

5.08 

2.53 

11 

3.36 

1.91 

Developing 

solutions 

13 

4.57 

2.50 

14 

4.07 

1.90 

13 

4.50 

2.93 

11 

1.64 

1.69 

Monitoring 
and evaluating 
a plan of 

13 

4.85 

1.99 

14 

3.00 

1.92 

13 

2.63 

2.64 

11 

1.18 

1.25 

action 














Note: The possible ranges of scores for problem representation are 0—7; developing solutions are 
0—6; monitoring and evaluating a plan of action are 0—6. 


Although there was no significant main effect on students’ conceptual knowledge 
among groups, we found that there was a significant main effect on ill-structured 
problem solving, Wilks Lambda (A) = .50, F( 9, 1 10) =4.0 ,p= .000. The 
multivariate r| 2 based on Cohen (1988) was quite strong, .21, and observed power 
also was considerably high, .97. Further, a univariate test of between-subjects 
effects revealed significant effects in two of the three problem solving processes, i.e. 
developing and justifying solutions, F (3, 47) =3.38, p=.G2G, r) 2 = . 18, observed 
power .73, and monitoring and evaluating a plan of action, F( 3, 47) =6.50, p=. 001, 
T) 2 = .29, observed power .96. Table 2 shows means and standard deviations for 
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the four conditions. The result indicated that the knowledge integration group 
had the highest mean scores on developing and justifying solution(s) (M=4.57, 
SD=2.50), and monitoring and evaluating a plan of action (M=4.85, SD=1.99) 
of all the groups. Follow-up post hoc comparison indicated significant 
differences between the KI and Control groups resided on developing/ 
evaluating solutions and monitoring/ evaluating a plan of action. 

Qualitative Findings 

The participants’ problem solving performance in different conditions were 
examined qualitatively to supplement the quantitative findings. Below is a brief 
report of students’ conceptual knowledge and ill-structured problem solving on 
the effects of different question prompts. 

Analysis of the participants’ ill-structured problem solving reports produced 
the following two findings. First, the participants who received knowledge 
integration prompts made intentional efforts to identify major concepts and 
their relationships that are necessary for solving the ill-structured problem. 

For example, “Validity refers to the appropriateness of the interpretation of 
the results of an assessment procedure for a given group of individuals, not to 
the procedure itself;” “the project was not valid as well when appropriateness 
of interpretation of assessment result... ;”and “the purpose of reliability is 
to ensure that the tests that are given are consistent and do not vary in their 
form of results...” On the other hand, participants who did not receive 
knowledge integration prompts appeared to identify the concepts vaguely and 
ambiguously, and they also failed to draw clear connections and relationships 
among concepts. Examples from students’ problem solving reports indicate 
that they used the concepts’ reliability and validity interchangeably and showed 
misconceptions of what the concepts mean: “A reliable and valid rubric will 
yield more accurate content assessment;” “if there were a clear set of criteria 
there could be more reliability since . . .;” and “. . .you might want to compare 
students grades to another to check validity.” 

Second, the participants who received problem solving prompts appeared 
to make efforts to construct solutions and explicitly provide arguments for 
subsequent implementation. Those prompts also directed the participants’ 
attention to seek alternative solutions they might have overlooked. For example, 
“Perhaps you can find another way to assess students, like have them sit in 
groups to work on things. However, make the projects individually based and 
grade each student individually upon completion of the project.” And another 
student said, “I suggest the teacher give objectives to the students so they know 
exactly what they are being evaluated on and can work toward that goal.” 
Comparing groups that received problem solving prompts with those that 
received knowledge integration prompts, the researcher found that problem 
solving prompts did not suffice to help the participants to retrieve constructed 
schema, which required participants to rely heavily on their limited working 
memory capacities to negotiate meanings. 

Lastly, the follow-up interviews also were analyzed to provide in-depth 
insights. The participants who received knowledge integration prompts said that 

368 Summer 2007: Volume 39 Number 4 

Copyright © 2007, ISTE (International Society for Technology in Education), 800.336.5191 
(U.S. & Canada) or 541.302.3777 (Int’l), iste@iste.org, www.iste.org. All rights reserved. 



those prompts helped them think about different aspects and the relationship of 
things. Those prompts also served as an “organizer” to understanding concepts 
and problem solving, and activated reflective thinking to draw attention to 
important elements, and also organized and transferred concepts as they applied 
in real-world situations. The participants who received only problem solving 
prompts mentioned that the problem seemed vague at first and, despite the 
prompts, they still had to spend time to pull the information together, relying 
more on their knowledge of the concepts than on problem solving process. 

One student mentioned, “I think they helped more in the problem-solving skill 
than the actual understanding of the concepts.” The problem solving prompts 
seemed to help students focus on some of the important problems in the case 
studies, analyze the specific problem in-depth, and break down the problems 
into sub-problems. 

DISCUSSION 

Although the quantitative results did not show a significant difference 
between students who received knowledge integration prompts and those who 
did not with regard to conceptual knowledge, the qualitative results indicated 
that these students felt that they developed a better integrated understanding 
of educational measurement despite the challenges they faced learning this 
topic. When the students were prompted to proceed with the knowledge 
integration process, they abandoned single knowledge elements for multiple 
knowledge elements, so that new or stronger connections were fostered through 
engagement with this Web-based learning environment. By the end, these 
students were more likely to explain their reasoning with a well-integrated 
answer than with a less connected answer. Clearly, compared to instruction 
without scaffoldings, providing scaffoldings was more beneficial. In many 
cognitive situations with instructional goals such as learning a new unit of 
domain knowledge, knowledge integration prompts may be more useful than 
problem solving prompts when asking students to engage in connecting, 
summarizing, and reflecting. Problem solving prompts, on the other hand, 
focus students’ attention only on the problem solving processes, and may not 
always correspond as well to students’ understandings. 

In the present study, problem solving prompts did not have a positive 
effect on solving the ill-structured problem. This finding contradicted Ge 
and Land’s (2003) study, in which problem solving prompts did not have 
effects on students’ ill-structured problem solving processes such as problem 
representation, developing and justifying solutions, and monitoring and 
evaluating a plan of action. In the qualitative analyses of students’ problem 
solving reports and follow-up interviews, it was found that these prompts 
directed students to construct solutions and argumentations for subsequent 
implementation. However, the absence of integrative knowledge structures 
caused them to overlook a broader range of solutions. 

The scaffoldings of both knowledge integration prompts and problem solving 
prompts did not have the increased positive results over the single scaffolding as 
we would expect. The knowledge integration prompts used in this study were 
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developed to help students identify weakness in their knowledge (Linn, 1995). 
The problem solving prompts were to guide students’ attention to perform like 
experts in the problem solving processes (Ge & Land, 2004). However, other 
research shows that when students cannot interpret specific prompts, they may 
flounder or dismiss the purposes of all those prompts (Davis, 2003; Ge, Chen, 

& Davis, 2005). Further, when students are not in control of their learning, 
they may experience fatigue that could greatly impact the overall performance. 
The use of question prompts as scaffolding should be used with caution and 
consideration of students’ prior knowledge and experiences. Excessive question 
prompts may cause a certain degree of cognitive overload, as seems indicated by 
the lower scores of participants with both knowledge integration and problem 
solving prompts. The design strategy of combining different types of prompts 
should be carefully evaluated in order to provide more explicit opportunities to 
help students apply concepts to real-world problems. A recent research endeavor 
looking at adaptive scaffolding by a human tutor provides new perspectives on 
how we can better diagnose student’s level of understanding (Azevedo, Cromley, 
& Seibert, 2004; Azevedo, Cromley, Winters, Moos, & Greene, 2005). 

IMPLICATIONS 

The study has practical implications for Web-based instruction, where 
question prompts embedded in Web-based learning environments can challenge 
students’ knowledge integration and support complex learning tasks. Web- 
based learning has considerable promise in the development and facilitation of 
students’ understanding. This study yields several implications for designing 
such an environment to support the development of cognitive skills and models 
of understanding. 

First, in order to support cognitive development effectively, a system is 
needed to facilitate intentional reflection and retention. Such a mechanism 
will raise students’ awareness of the gaps and detect biases in their knowledge 
bases. Therefore, the Web-based cognitive modeling system should structure 
opportunities for intentional reflection. 

Second, in order to engage students in meaningful learning, the system 
should provide real-world problems to help students reconcile the application 
of knowledge. Students often experience difficulty connecting educational 
theories with real-world problems; therefore, utilizing question prompts to act 
as a mediator for the instructor role will enable the linking system to activate 
students’ knowledge application. 

Last, but not least, it may be useful to provide feedback or features that 
structure opportunities for students to interact with experts and their peers. 
Distance learning is characterized by the separation between instructors and 
the students, whether by temporal or spatial distance. The cognitive advantage 
of incorporating a system that includes knowledge integration and problem 
solving prompts into distance education is that viewing materials from 
multiple perspectives can increase cognitive flexibility and interconnections as 
well as giving opportunities to share and become involved with the learning 
community. 
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RECOMMENDATIONS FOR FUTURE RESEARCH 

This study urges further validation on the assessment of knowledge 
integration. Multiple assessment sources should help define and better 
represent knowledge integradon. Possible multiple assessment sources should 
be investigated. Accordingly, it may be beneficial to examine and adapt 
methods used in other fields to understand knowledge integration better. For 
instance, a group of researchers used benchmarking as one of the techniques 
to assess the process of knowledge integration (Lee, Husic, Liu, & Hofstetter, 
2006). A mixed initiative assessment that includes humans and computers has 
been widely discussed and implemented in real scaffolding classroom settings 
(e.g., Zimmerman, 2005). This dynamic assessment evaluates transitions in 
knowledge representations and performance while learners are in the process 
of solving problems, rather than after they have completed a problem, which is 
critical for providing valid interpretations of students’ performance. A further 
investigation on the use of computers to assess and capture students’ learning 
progress needs to be explored and discussed. 

Another avenue for future research is how knowledge is integrated through 
the use of other scaffolding techniques, such as cooperative groups and expert 
modeling (Lajoie, 2005). Peer learning is assumed to promote sharing and 
the development of understanding. This study indicates that interaction and 
feedback loops may be useful in facilitating better knowledge integration. The 
Web-based learning environment developed for use in this study was specifically 
designed to promote knowledge integration and problem solving through 
different types of question prompts. From a design perspective, it would be 
useful to know whether other tools had a greater impact on students’ knowledge 
integration than the question prompts (Chen & Ge, 2006). Knowing this 
might lead to more generalizable design principles for designing better 
knowledge integration environments. Research on this question would likely 
involve conducting interviews with students specifically regarding the use of 
tools, collecting pre-post analyses of knowledge integration, tracking students’ 
progress (e.g., time logs of overall frequency of use as well as frequency of 
specific tool use), and increasing numbers of participants in the treatment and 
control groups to provide additional valuable information. 
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